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LINEAR PREDICTION OF STATIONARY VECTOR SEQUENCES 


Yoram Baram* 


SUMMARY 


The class of all linear predictors of minimal order for a stationary vector- 
valued process is specified in terms of linear transformations on the associated 
Hankel covariance matrix. Two particular transformations, yielding computationally 
efficient construction schemes, are proposed. 


1 . INTRODUCTION 


The prediction of a vector-valued stationary sequence possessing a linear 
representation of finite order can be approached, in principle, by first construct- 
ing such a representation from the process second order statistics and then matching 
a Kalman filter to it. The construction of a linear representation can be performed 
by spectral factorization, as suggested by Anderson [1] or by stochastic realiza- 
tion, as suggested by Faurre [2]. Given the covariance function of the process in 
the time domain, both approaches would be indirect, as the first requires the trans- 
formation to the frequency domain, while the second requires the intermediate solu- 
tion of a different problem, namely, the deterministic input-output realization 
problem. The construction of the Kalman filter further requires the solution of a 
Riccati equation. More direct predictor constructions from the covariance function, 
suggested by Faurre [3] and by Son and Anderson [4], also require the solution of 
Riccati equations. 

In this paper we propose a direct approach to the construction of linear pre- 
dictors for stationary vector sequences from the covariance function. The approach, 
inspired by Akaike's coordinate-free realization concepts [5], is based on simple 
geometric principles. It suggests an explicit coordinate-dependent characterization 
of the class of all minimal order linear predictors, in terms of linear transforma- 
tions on the Hankel covariance matrix associated with the sequence. It does not 
give rise to Riccati or Lyapunov equations. The selection of particular transforma- 
tion matrices defines specific predictor construction techniques. 


*Y. Baram is a Senior Research Associate of the National Research Council at 
the NASA Ames Research Center, Moffett Field, CA 94035, on sabbatical leave from the 
Department of Electrical Engineering, Technion, Israel Institute of Technology, 

Haifa 32000, Israel. 
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2. GENERALIZED FORMULATION 


Let the covariance function of a zero-mean, full rank stationary sequence 
e r"" be given by 


<2.i) 

where E denotes the expectation operation, and suppose that there exists a posi- 
tive integer P and a set of scalars a.],..., ap_-|, aQ = 1 , such that for any n > P 

P-1 

Z Wk = ° 

k=0 


It is well known that when y^ is generated by a finite dimensional linear system 
driven by white noise, a relationship of the type (2.2) is satisfied. 

The prediction problem is one of estimating the values of yn+k> k = 0,1,..., 

given the values of y^.,, yn-2,... . Let us denote by Y' = (y^ ,,y'^ 

+ T T T 

and Y = (y ,y the past and the future vectors at time n. Let y., 

n •'n’-'n+r •'k,! 

denote the ith element of yj^ and let denote the linear mean-square 

projection of y^^ on Y~. Let us further denote by the space generated by 

‘ykaln-r^ = ' >'Lk>’^ 

+ T T T T 

Y (k) = (y ,y y , ,) the k-step past and future vectors at time n and 

n n n+] n+k-i 

by ^n|n-l^^^ linear mean-square projection of y*(k) on Y~. We have 


where 


«1„- 


Y„|„.,(k) = R(k,n)(E(Y„Y„ )] ’Y„ 


R(k,n) = E(Y*(k)Y;;) = 
n n 


\+i 


n+1 


W-1 
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It follows from (2.2) that for any k > P the rows beyond the first P • m rows 
of R(k,n) are linearly dependent on the previous ones. The space (Y^|Y~) is then 
spanned by 

T . 

Y'^i ,(P) = R(P,n)[E{Y"Y‘ }]~'y" 
n|n-1 ' » /I n P j n 

It can also be seen from (2.2) that the columns of R(k,n) beyond the first P • m 
columns are linearly dependent on the previous ones. It follows that the first 
maximal set of linearly independent rows of R(P,n) is indexed as the set of such 
rows of the matrix 


R = E(Yjj(P)[Y~(P)]‘^} 


The subsequent analysis will require some further notation. Let A and B be 
any two matrices such that A has no fewer rows and no fewer columns than B. Let 
us denote by matrix whose rows are those rows of A, indexed as the first 

maximal set of linearly independent rows of B. Similarly, let us denote by Cg(A) 
the matrix whose columns are those columns of A indexed as the first maximal set 
of linearly independent columns of B. We will find it convenient to write these 
row and column selection operations in terms of matrix products 


R^ R2 

R2 R3 


Rp Rp^l 


P+1 


2P-1 


r*B(A) = r^A 
and 

ft! 

Ob(A) i ACj 


A A 

where rg and Cg are properly dimensioned, full rank matrices of zeros and ones. 
For instance, suppose that A has dimension 5^7 and that rows 1, 3, and 4 and 
columns 1, 2, and 5 of B form maximal independent sets. Then 


r 


A 

B ■ 


1 

0 

0 


0 0 
0 1 
0 0 


0 0 
0 0 
1 0 


I 
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and 


10 0 0 0 0 0 
0 1 0 0 0 0 0 
0 0 0 0 1 0 0 


Since it may be assumed that the matrices involved have compatible dimensions, we 
will use the somewhat abbreviated notation 


rg = r 


A 

B ’ 


A 


Let M and N be arbitrary nonsingular square matrices of dimension m*P and 
let 


and 



(P) 


= ny'cp) 

n n 


Let us further define 


and 


= r 


MRN 


T^n 


zV , 

" MRN^ 


It can be seen that is a vector of minimal dimension which spans (^^1^") in the 

sense that each element of the latter space can be obtained by a linear combination 
of the elements of x^. Clearly, x^^^^ belongs to which is included in 

(Y^|y“^l). The latter may be decomposed as (^^l^n+l^ " ^^n^^n^ ® '^n '^n 

the space generated by 


V 

n 



- y 


n |n-1 
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and ® denotes the Cartesian product. Since is orthogonal to (Yj^|Y^), it 

follows that there exist matrices and such that 


X , = A X + B V 
n+1 n n n n 




Multiplying (2.4) on the right by taking expectation and noting that 

is orthogonal to Y~, hence, to 2 .^, we obtain 


E{x .z"^} = A E{x z”^} 
n+1 n n n n 


(2.5) 


where 


E(x z-J) = r t”E{yJ-| .(P)[Y"(P)]'^}nV „ 
" " MRN^ " MRN^ 


T T 

r MRN^c^ 
MRN MRN 


= MRN^ 


(2.6) 


where 

M = r M , N = c N 
MRN^ MRN 

T 

It can be seen that E{x z } is a non-singular matrix. We also have 

n n 


^ME{Y„.,,„(P)[y:(P)]'^}nV 


MRN 


T"“=-'S+1|n' 


MRN 


= MR^N^ 


(2.7) 


where 


R^ = E{Y*^^(P)[Y‘(P)]‘^} = 


R., R- 


^P+1 ^P+2 


P+1 

^P+2 


'2P 


( 2 . 8 ) 
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Substituting (2.6) and (2.7) into (2.5), we obtain 

A = A = MR®N*^(MRN^)"^ 
n 

It follows from the definition of that there exists a matrix such 

that 


y„ = C x„ + 

•'n n n n 


(2.9) 


Multiplying (2.9) on the right by and taking expectation, we obtain 




( 2 . 10 ) 


where 

E{y z^} = (MRn’^) 

where (MRN^)^ is the first block row of MRN^. Hence, 


C = C = (mrn'^).c'^ -.(mrn”^)'^ 

MRN 


Multiplying (2.4) on the right by and taking expectation, noting that is 

orthogonal to x^^, we obtain 


where 


and 


= MR - Anc*^ 

E(v„vT, = E((y„ - Cx„)yT) 

= R - cnc*^ 
o 


( 2 . 11 ) 


( 2 . 12 ) 


(2.13) 
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where 


and 


R = 


n = E{x„x„"} 
n n 


(fi{ 4 ■■■ «y 


(2.14) 


We next derive the term nC , which appears in both (2.12) and (2.13). Noting that 




it can be seen that 


one = R 


Where R is defined by (2.14) and 


0 = 


’CA 

CA‘ 


CA" 


(2.15) 


Since ^n|n-1^^^ " since the covariance rank of x^^ is the same as that 

of Yn|n-1^^^* follows that 0 is full rank. Hence, 


nC^ = (0^0)~^0^R 


which, substituted in (2.12) and (2.13), completes the derivation of the matrix B, 
defined by (2.11). In summary, the generalized predictor is given by 


x^ . = Ax + Bv , X = 0 
n+1 n n ' o 


y , I , = CA^x 
■^n+k|n-1 n 


( 2 . 16 ) 
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where 


(2.17) 


— <?— T T -1 

A = MR*^N (MRN ) 

C = (MRn'^) ™(MRN^)“'' 

' MRN^ 

B = [M - A(0^0)‘^O^]R[R^ - C(0'^0)“^0^R] 

and 


(2.18) 

(2.19) 


The class of all minimal order linear predictors for the sequence yj^ is now 
defined by (2.16). A particular member of the class is specified by selection of 
the nonsingular matrices M and N. Two particular selections are suggested in the 
following section. We note that 


n+1 


= Ax + Bv 


yn = Cx, 


( 2 . 20 ) 


With A, B, C, and as defined above, is a minimal realization of the sequence 

yn- 


3. SPECIFIC CONSTRUCTIONS 


A. Original Coordinates 

When the matrices M and N are taken to be identity matrices, the vectors 
Y~(P) and Y^(P) are left in their original coordinates. The minimal predictor for 
tRis choice^is specified in the preceding section, with MRN^ and MR®N^ replaced 
simply by R and R®. The predictor is essentially defined by maximal sets of lin- 
early independent rows and linearly independent columns of R. The following proce- 
dure selects a maximal set of such rows. Extension to the selection of linearly 
independent columns is immediate. 

The procedure consists of checking the rank deficiency of matrices R^^, con- 
sisting of the first i-1 linearly independent rows of R and the row following 
them. If Rj^ is rank-deficient, its last row is replaced by the next row of R 
and the rank check is repeated. If R^^ has full-column rank, the next row of R 
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is attached to Rj to form and a rank check is performed. This procedure is 

continued until rank = M or until all the rows of R have been checked. The 
final R^ will contain a maximal set of linearly independent rows of R. This 
procedure can be extended to the construction of a linear predictor or a minimal 
realization from a sample covariance sequence, employing a recently proposed statis- 
tical rank test method [6]. 


B. Canonical Coordinates 

Construction in the original coordinates requires a sequence of rank checks on 
submatrices of the Hankel covariance matrix R. A computationally more attractive 
technique may be obtained by selecting the matrices M and N so as to diagonalize 
the matrix MRN^. Let M and N be the matrices of normalized eigenvectors of RR^ 
and R^R, respectively, corresponding to the nonzero eigenvalues. These may be 
obtained directly from the singular value decomposition of R (see, e.g., [7]). It 
can be readily seen that 


r m c c .j, — 

MRN MRN 


Denoting 


(3.1) 


S = 


’PJ 


where Sp...,Sp are the squared nonzero singular values of R, the predictor is 
now specified by the matrices 


A = [Ip 0]MR®N^ 


I, 




(3.2) 


C = [s^ 0 ••• 0]S 


-1 


(3.3) 


and B is defined by (2.19), with r - defined by (3.1), © defined by (2.15), 

MRN 

and A and C defined by (3.2) and (3.3). We note that while the above procedure is 
computationally more effective than the one performed in the original coordinates, 
when the exact covariance function is known, it is not presently extendable by means 
of statistical inference to the case where only a sample covariance sequence is 
given. We also note that the above construction is different from the canonical 
variates realization method suggested by Larimore [8], which produces an approximate 
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representation for the process, even when the latter possesses a finite order reali 
zation. The present construction produces an exact realization of the form (2.20) 
with the parameters specified above. 


4. CONCLUSION 


A direct approach to the construction of a minimal order linear predictor for 
stationary vector sequence from its covariance function has been proposed. The 
class of all such predictors has been characterized in terms of linear transforma- 
tions on the Hankel covariance matrix. Two specific construction schemes have been 
specified. 
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